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Abstract  for  SF  298:  New  mathematical  theory  and  algorithms  were  developed  for 
querying  functions  and  data  sets  in  high  dimensions.  This  was  accomplished  in  both  the 
deterministic  setting  as  well  as  the  stochastic  setting  with  noise.  Since  high  dimensional 
problems  suffer  from  the  curse  of  dimensionality,  new  model  classes  were  introduced  based 
on  the  notions  of  sparsity  and  variable  reductions.  These  model  classes  were  shown  to  fit 
real  world  problems  and  yet  be  amenable  to  realistic  computational  methods  for  query¬ 
ing  in  high  dimensions.  The  new  algorithms  were  developed  using  sophisticated  adaptive 
methods.  They  utilize  the  most  judicious  deterministic  point  clouds  in  high  dimension 
such  as  those  based  on  discrepancy  theory  (Halton  sequences)  and  perfect  hashing.  The 
resulting  algorithms  were  shown  to  have  optimal  performance  on  the  proposed  model 
classes.  Thus,  they  have  provided  the  most  efficient  algorithms  for  querying  high  dimen¬ 
sional  data  under  the  model  assumptions.  Results  were  also  developed  and  proved  for 
classification  of  data.  These  stochastic  algorithms  were  shown  to  have  optimal  perfor¬ 
mance  on  various  model  classes. 


Final  Report:  Modern  data  processing  is  challenged  by  the  size  and  variety  of  data 
that  must  be  queried  to  extract  the  relevant  details  needed  to  answer  targeted  questions. 
Mathematically,  the  problem  can  be  formulated  as  learning  a  function  depending  on  a 
large  number  of  variables.  The  type  of  learning  problem  encountered  depends  on  the 
information  available  about  the  function.  In  directed  learning  we  are  allowed  to  query  the 
function  at  any  points  we  wish  but  at  perhaps  a  very  high  cost  and  the  question  is  where 
should  we  direct  our  queries.  In  stochastic  learning,  the  values  are  given  to  us  as  random 
draws  (usually  with  noise)  and  the  question  is  how  to  best  utilize  this  information. 

The  inherent  impediment  to  learning  in  high  dimension  is  the  so  called  curse  of  di¬ 
mensionality  which  says  our  ability  to  recover  a  smooth  function  deteriorates  significantly 
with  increasing  dimension.  The  only  way  around  this  is  if  the  target  functions  satisfies 
additional  properties  which  reduce  their  complexity.  The  problem  is  to  identify  math¬ 
ematically  what  these  properties  may  be  and  then  utilize  them  in  the  development  of 
learning  and  computation  algorithms.  The  goal  of  the  research  project  is  to  develop  new 
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model  classes  which  represent  real  world  problems  in  high  dimension,  yet  do  not  suffer  the 
curse  of  dimensionality.  These  models  are  based  on  the  view  that  only  a  small  number 
of  the  variables  are  significant  and  if  these  can  be  found  then  a  model  reduction  can  be 
performed  to  make  the  learning  problem  more  tractable.  Its  main  focus  is  to  develop  new 
model  classes  in  high  dimension  that  give  a  precise  description  of  this  variable  reduction 
and  then  to  develop  algorithms  that  perform  at  optimal  distortion  in  terms  of  the  com¬ 
putational  budget  for  each  of  these  new  classes.  The  PI  and  his  collaborators  have  put 
forward  several  model  classes  for  functions  of  many  variables  and  shown  how  to  develop 
optimal  query  algorithms  for  each  of  these  model  classes.  In  addition  these  models  use 
sparsity  and  multiscale  expansions  to  further  increase  their  tractability.  The  following  is 
a  summary  of  the  notable  achievements  under  this  research  project. 


Directed  learning  in  high  dimensions  With  Guergana  Petrova  and  Przemek  Woj- 
taszczyk,  the  PI  has  investigated  the  problem  of  capturing  a  continuous  function  /  on  a 
compact  domain  C  1RN  with  N  large  by  querying  the  point  values  of  /.  This  problem 
is  sometimes  called  directed  learning.  It  is  known  that  this  problem  suffers  from  the  curse 
of  dimensionality  which  means  that  even  smooth  functions  need  an  impossible  number 
of  queries.  For  example,  for  a  general  function  in  Cs,  we  would  need  e~N^s  queries  to 
capture  it  to  accuracy  e.  The  only  way  to  defeat  this  curse  and  have  reasonable  numerical 
algorithms  is  to  assume  more  about  /.  The  dominant  theme  in  applications  is  to  assume 
that  /  actually  depends  on  much  fewer  variables  (unknown  to  us)  or  it  can  be  well  ap¬ 
proximated  by  functions  of  few  variables.  In  our  work  [10],  we  have  shown  that  under 
both  of  these  models  we  can  break  the  curse  of  dimensionality  by  tailoring  our  questions. 
This  work  is  very  foundational  and  of  interest  also  in  computer  science  and  statistics. 

The  work  described  above  has  captured  the  interest  of  several  other  researchers.  With 
Albert  Cohen  ,  Ingrid  Daubechies,  Gerard  Kerkyacharian,  and  Dominique  Picard  [6],  the 
PI  has  extended  the  results  of  [10]  to  include  other  models  for  functions.  The  simplest 
examples  are  the  ridge  functions  f(x)  =  g  (a-  x).  We  show  how  to  capture  such  a  function 
/  with  g  and  a  unknown  by  querying  its  point  values  at  specially  constructed  point  sets. 
Our  approach  is  a  combination  of  hashing  and  compressed  sensing  techniques.  Our  first 
versions  of  these  algorithms  were  not  stable  in  terms  of  perturbations  of  the  queries.  We 
have  recently  uncovered  how  to  handle  the  instability. 

Stochastic  learning:  We  revisited  the  fundamental  problem  of  classification  from  ran¬ 
domly  drawn  data  with  a  particular  eye  towards  high  dimensional  problems  [3] .  We  have 
developed  new  ways  to  bound  variance  in  terms  of  VC  dimension.  We  also  developed 
new  model  classes  for  the  regression  function  which  reflect  properties  which  enable  fast 
algorithms  even  in  high  dimension.  One  difficulty  in  treating  high  dimensional  problems 
is  that  standard  localization  techniques  such  as  adaptive  partitioning  are  impossible  to 
implement  in  their  standard  form  because  one  adaptive  refinement  results  in  an  inordinate 
number  of  cells.  As  an  alternative,  we  have  developed  with  Peter  Binev  and  Wolfgang 
Dahmen  a  new  theory  for  adaptive  partitioning  called  sparse  occupancy  trees.  The  cardi¬ 
nality  of  these  trees  is  controlled  by  the  initial  size  of  the  data.  Therefore  they  become 
numerically  feasible  since  the  number  of  computations  is  linear  in  the  size  of  the  data. 
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This  has  allowed  us  to  develop  learning  algorithms  in  [5]  for  regression  problems  in  high 
dimension  which  have  several  advantages  over  the  existing  techniques  based  on  nearest 
neighbors  or  support  vector  machines.  A  follow  up  giving  an  analysis  of  the  numerical 
performance  of  these  new  classification  algorithms  is  given  in  [4] 

Querying  manifolds  in  high  dimension:  One  of  the  common  ways  of  avoiding  the 
curse  of  dimensionality  is  to  assume  that  the  data  comes  from  a  smooth  low  dimensional 
manifold  in  high  dimensions.  We  have  developed  greedy  algorithms  for  taking  snapshots 
of  these  manifolds  and  then  using  them  to  create  good  linear  space  fits  to  the  manifold  [2], 
We  have  been  able  to  accomplish  this  even  in  the  setting  of  infinite  dimensional  Banach 
spaces  [11].  These  techniques  are  heavily  used  in  model  reduction  for  PDEs. 

Stochastic  and  parametric  PDEs:  Such  PDEs  model  many  physical  systems.  The 
standard  approach  to  solving  stochastic  PDEs  are  Monte  Carlo  methods.  In  [7],  we  offer 
an  attractive  alternative  to  Monte  Carlo  for  solving  elliptic  stochastic  equations.  Our 
methods  are  based  on  Wiener  chaos  expansions  which  convert  the  stochastic  equations  to 
parametric  equations  with  an  infinite  number  of  parameters.  We  show  that  the  resulting 
parametric  equations  have  a  sparse  representation  with  respect  to  polynomial  basis  in  the 
parameters.  This  lays  the  foundation  for  efficient  numerical  methods  to  solve  the  original 
stochastic  equation.  They  solve  infinite  dimensional  parametric  problems  in  polynomial 
time. 

This  original  work  gave  dramatic  improvement  in  computational  efficiency  over  Monte 
Carlo.  However,  some  of  the  assumptions  on  the  diffusion  coefficients  were  not  natural. 
In  [8],  we  introduce  complex  variable  methods  to  obtain  what  seem  to  be  the  most  natural 
and  far  reaching  results  in  terms  of  efficiency  of  recovery. 

This  work  on  stochastic  PDEs  has  been  implemented  in  numerical  algorithms  but  not 
to  the  full  extent  of  the  theory.  With  Chkifa,  Cohen  and  Schwab  [9],  we  have  given  a 
greedy  procedure  which  will  generate  optimal  Wiener  chaos  polynomial  spaces  in  which 
to  capture  the  solution  to  the  stochastic  PDE.  These  algorithms  are  an  analogue  of  the 
famous  wavelet  greedy  algorithms  developed  by  Cohen-Dahmen-DeVore.  They  display 
dramatic  improvement  in  their  rate  distortion  over  all  existing  algorithms. 

Tensor  structure:  One  of  the  common  themes  to  handle  high  dimensional  structure 
in  quantam  chemistry  is  to  assume  a  tensor  structure.  There  are  many  possible  tensor 
formats  and  a  coherent  theory  for  approximation  by  tensors  has  not  yet  been  developed. 
We  have  given  first  steps  toward  such  a  theory  in  [1]  where  querying  algorithms  for  low 
rank  tensors  are  considered. 
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